Decision Tree Models Applied to the Labeling of Text with Parts-of-Speech

1. I n t r o d u c t i o n In this paper we describe work which uses decision trees to estimate probabilities of words appearing with various parts-of-speech, given the context in which the words appear. In principle, this approach affords the opt imal solution to the problem of predicting the correct sequence of parts-of-speech. In practice, the method is limited by the lack of large, hand-labeled training corpora, as well as by the difficulty inherent in constructing a set of questions to be used in the decision procedure. Nevertheless, decision trees provide a powerful mechanism for tackling the problem of modeling long-distance dependencies. The following sentence is typical of the difficulties facing a tagging program: The new energy policy announced in December by the P r i m e Min is ter will guarantee sufficien~ oil supplies at one price only. structed a complete set of binary questions to be asked of words, using a mutual information clustering procedure [2]. We then extracted a set of events from a 2-million word corpus of hand-labeled text. Using an algori thm similar to that described in [1], the set of contexts was divided into equivalence classes using a decision procedure which queried the binary questions, splitting the da ta based upon the principle of max imum mutual information between tags and questions. The resulting tree was then smoothed using the forward-backward algori thm [6] on a set of held-out events, and tested on a set of previously unseen sentences from the hand-labeled corpus. The results showed a modest improvement over the usual hidden Markov model approach. We present explanations and examples of the results obtained and suggest ideas for obtaining further improvements. 2. D e c i s i o n T r e e s The problem at hand is to predict a tag for a given word in a sentence, taking into consideration the tags assigned to previous words, as well as the remaining words in the sentence. Thus, if we wish to predict tag S,~ for word w~ in a sentence S -wl, w2, • • •, wN, then we must form an est imate of the probabili ty The usual hidden Markov model, trained as described the last section of this paper, incorrectly labeled the verb announced as having the active rather than the passive aspect. If, however, a decision procedure is used to resolve the ambiguity, the context may be queried to determine the nature of the verb as well its agent. We can easily imagine, for example, tha t if the bat tery of available questions is rich enough to include such queries as "Is the previous noun inanimate?" and "Does the preposition by appear within three words of the word being tagged?" then such ambiguities may be probabilistically resolved. Thus it is evident that the success of the decision approach will rely in the questions as well as the manner in which they are asked. In the experiments described in this paper, we conP(S,~ [ $1, $2,. . .S,~-1 and wl , w2 . . . , war). We will refer to a sequence ($1, . . . , t,~-l; w l , . . . , wN) as a history. A generic history is denoted as H, or as H = (HT, Hw), when we wish to separate it into its tag and word components. The set of histories is denoted by 7-/, and a pair (t, H) is called an event. A tag is chosen from a fixed tag vocabulary VT, and words are chosen from a word vocabulary Vw. Given a training corpus E of events, the decision tree method proceeds by placing the observed histories into equivalence classes by asking binary questions about them. Thus, a tree is grown with each node labeled by a question q : 7-/ --~ {True, False}. The entropy of tags at a